Using student experience as a model for designing an automatic feedback system for short essays

The SAFeSEA project (Supportive Automated Feedback for Short Essay Answers) aims to develop an automated feedback system to support university students as they write summative essays. Empirical studies carried out in the initial phase of the system’s development illuminated students’ approaches to and understandings of the essay-writing process. Findings from these studies suggested that, regardless of their experience of higher education, students consider essay-writing as: 1) a sequential set of activities, 2) a process that is enhanced through particular sources of support and 3) a skill that requires the development of personal strategies. Further data collected from tutors offered insight into the feedback and reflection stages of essay-writing. These perspectives offered a fundamental model of essay-writing and feedback to inform the ongoing, iterative development of this automated feedback system and indeed, for any institution developing tools to support students’ writing

Everyday language is highly intensional

Abstract There has recently been a great deal of work aimed at trying to extract information from substantial texts for tasks such as question answering. Much of this work has dealt with texts which are reasonably large, but which are known to contain reliable relevant information, e.g. FAQ lists, online encyclopaedias, rather than looking at huge unorganised resources such as the web. We believe, however, that even this work underestimates the complexity and subtlety of language, and hence will inevitably be restricted in what it can cope with. In particular, everyday use of language involves considerable amounts of reasoning over intensional objects (properties and propositions). In order to respond appropriately to simple-seeming questions such as 'Is going for a walk good for me?', for instance, you have to be able to talk about event-types, which are intrinsically intensional. We discuss the issues involved in handling such items, and shows the kind of background knowledge that is required for drawing the appropriate conclusions about them

OpenEssayist: real-life testing of an automated feedback system for draft essay writing

OpenEssayist is unique in being an automated feedback system that has been developed to offer feedback on students' draft essays, rather than assessment on their finished work. This is therefore a system that offers opportunities for students to engage with and reflect on their work, and to improve their work through understanding of the requirements of academic essay writing. In trialling use of the system in a genuine Open University course, we found that students made use of it to varying degrees, which is perhaps likely with any study resource. Those who took the time to explore system affordances and what they could be used for however tended to report more positively on its perceived value. From our analysis we were also able to conclude that a significant positive correlation exists in this sample of students between marks on essay 1 and the number of drafts submitted. We could speculate as to what this may mean for this set of students, or more widely, but it seems clear that use of a system such as OpenEssayist has many potential advantages to students and tutors, which will benefit from further research and exploration

What types of essay feedback influence implementation: Structure alone or structure and content?

Students approach educational courses with varying levels of experience and understanding, and so need appropriate support to inform them of expectations and to guide their learning efforts. Feedback is critical in this process, so that learners can gauge how their performance aligns with expectations, and how they can improve their efforts and attainments. This study focused on the effects of providing different types of feedback on participants’ written essays, as well as on participants’ motivations for learning using measures of motivation and self-efficacy. In terms of research questions, it was important to ascertain whether participants performed differently in subsequent essays after receiving feedback on structure alone or on structure and content; whether their self-reported levels of motivation and attitudes to learning were related to essay performance; and whether the difference in type of feedback affected their self-reported levels of motivation and attitudes to learning. Findings revealed no significant difference in marks between those receiving feedback on structure alone and those receiving feedback on structure and content, which is surprising and deserves further exploration. Even so, using feedback to highlight certain structural elements of essay writing can have a lasting positive impact on participants’ future essay performance

OpenEssayist: extractive summarisation and formative assessment of free-text essays

OpenEssayist is a system which is currently under development. It aims to provide an effective automated interactive feedback system that yields an acceptable level of support for university students writing summative essays. The principal natural language processing technique currently employed is extractive summarisation using graph-based ranking algorithms. OpenEssayist will be piloted in September 2013 with Open University UK students following a Master’s course of study

An exploration of the features of graded student essays using domain-independent natural language processing techniques

This paper presents observations that were made about a corpus of 135 graded student essays by analysing them with a computer program that we are designing to provide automated formative feedback on draft essays. In order to provide individualised feedback to help students to improve their essays, the program carries out automatic essay structure recognition and uses domain-independent graph-based ranking techniques to derive extractive summaries. These procedures generate data concerning an essay’s organisational structure and its discourse structure. We have selected 27 attributes from the data and used them in a comparative analysis of all the essays with a view to informing further development of the feedback program. The results of this analysis suggest that some characteristics of students’ essays that our domain-independent feedback program is measuring may be related to the grades that tutors assign to their essays

Designing and testing visual representations of draft essays for higher education students

This paper reports the findings of an empirical investigation, which set out to test a set of rainbow essay exercises. The rainbow diagrams are pictorial representations of formal graphs that are derived automatically from student essays. They were designed to allow students to discover how key concepts in a well written essay are connected together. The students would then be able to compare a rainbow diagram of their own essay with a good essay and make changes to it before submission to their tutor. However a trail was undertaken with academics, teaching and learning staff, doctoral students at the Open University of Catalonia and the Open University UK, before implementation into the web application known as Open Essayist. All the participants from each University completed the exercise correctly. This was a surprising finding as we expected participants to experience some difficulties, as previous visual representations we piloted. All the participants remarked that they had learnt a lot about the structure of good essays and more importantly how clear the role of the conclusion played in a well-constructed essay. This type of representation made this explicit and they would be able to see quickly if a second draft had improved. The users also mentioned that the rainbow diagram representations could be used as a generic essay feedback tool. It could be used across subject domains, a hypothesis worthy of further investigation

OpenEssayist: an automated feedback system that supports university students as they write summative essays

OpenEssayist is an automated, interactive feedback system designed to provide an acceptable level of support for students as they write essays for summative assessment. There are two main components to the system: (1) a linguistic analysis engine and (2) a web application that generates feedback for students The main pedagogical challenge in the e-assessment of free text is how to provide meaningful “advice for action” in order to support students writing their summative assessments. We have built a first working version of the system in which we use unsupervised graph-based ranking algorithms (following Mihalcea & Tarau, 2005) to automatically extract key words, phrases and sentences from student essays. We have developed several external representations of these summarisation techniques. For examples, key words and key phrases can be viewed in a word cloud or in a dispersion graph, and they can be explored and organised into groups. Holistic approaches have also been tested using ‘mash ups’ where key words and key sentences are highlighted in context in the essay itself, helping students to investigate the distribution of key words and its potential implications for the clarity of the narrative. This paper will report the findings from our pilot studies of the interactive models associated with the summarisation techniques

A prototype for a conversational companion for reminiscing about images

This work was funded by the COMPANIONS project sponsored by the European Commission as part of the Information Society Technologies (IST) programme under EC grant number IST-FP6-034434. Companions demonstrators can be seen at: http://www.dcs.shef.ac.uk/∼roberta/companions/Web/.This paper describes an initial prototype of the Companions project (www.companions-project.org): the Senior Companion (SC), designed to be a platform to display novel approaches to: (1) The use of Information Extraction (IE) techniques to extract the content of incoming dialogue utterances after an ASR phase. (2) The conversion of the input to RDF form to allow the generation of new facts from existing ones, under the control of a Dialogue Manager (DM), that also has access to stored knowledge and knowledge accessed in real time from the web, all in RDF form. (3) A DM expressed as a stack and network virtual machine that models mixed initiative in dialogue control. (4) A tuned dialogue act detector based on corpus evidence. The prototype platform was evaluated, and we describe this; it is also designed to support more extensive forms of emotion detection carried by both speech and lexical content, as well as extended forms of machine learning. We describe preliminary studies and results for these, in particular a novel approach to enabling reinforcement learning for open dialogue systems through the detection of emotion in the speech signal and its deployment as a form of a learned DM, at a higher level than the DM virtual machine and able to direct the SC’s responses to a more emotionally appropriate part of its repertoire. © 2010 Elsevier Ltd. All rights reserved.peer-reviewe

The senior companion : a semantic web dialogue system

This work was funded by Companions[3], European Commission Sixth Framework Programme Information Society Technologies Integrated Project IST-34434.7The Senior Companion (SC) is a fully implemented Windows application intended for intermittent use by one user only (a senior citizen) over potentially many years. The thinking behind the SC is to make a device that will give its owner comfort, company, entertainment, and some practical functions. The SC will typically be installed at home, either as an application on a personal computer, or on a dedicated device (like a Chumby) or an intelligent coffee table (like Microsoft's Surface). By means of multimodal input and output, and a graphical interface, the SC provides its 'owner' with different functionalities, which currently include: • conversing with the user about his personal photos • learning about the user, user's family, and life history • telling the user jokes • reading the news (via RSS feed from the internet)peer-reviewe